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Foreword 


Monitoring Transparency in Higher Education 


‘The assessment bandwagon has been rolling for decades. But it wasn’t until six years ago when U.S. 
Secretary of Education Margaret Spellings impaneled the National Commission on the Future of Higher 
Education that outcomes assessment caught the attention of the popular media as well as a critical mass 
of policy makers and mainstream college and university leaders. The Commission proposed six sweeping 
recommendations, one of which was to create a “consumer-friendly information database” so that 
prospective students ostensibly could find answers to the kinds of questions one would ask when making 
a major purchase: 


¢ How much does it cost? 

¢ How do I finance the purchase? 

¢ How good is this “product” relative to others? 
¢ What do the performance data show? 


This last topic — providing comparable evidence of performance — hadn't gotten much traction, even 
among those who were proponents of outcomes assessment. But it sure got a spike in attention as a 
result of the Commission’s bully pulpit. 


Since that time, several leading institutional membership organizations developed templates for their 
member colleges and universities to publicly report the activities and evidence on their campus related 
to student academic attainment. Balancing the legitimate demands of public interest and institutional 
autonomy, the objective was to test the appropriate and acceptable applications of transparency efforts 
and ameliorate problematic aspects of making data public. By all accounts, these initial steps seem to 
have been at least modestly successful. 


For example, at this point almost 300 universities have agreed to experiment with the College Portrait 
segment of the Voluntary System of Accountability championed by the Association of Public and Land- 
grant Universities (APLU) and American Association of State Colleges and Universities. In the 
independent sector, the National Association of Independent Colleges and Universities initiated the 
University and College Accountability Network (U-CAN). NAICU also recently launched another effort 
in cooperation with the Council of Independent Colleges (CIC), Building Blocks to 2020, which is a 
searchable web database that includes a variety of measures including persistence and graduate rates 
and results from such measures as student surveys and standardized tests. The most recent transparency 
template to appear is the Voluntary Framework of Accountability being tested by the Association of 
American Community Colleges with institutions in that sector. 


While NILOAs national surveys suggest the attention devoted to assessing student attainment continues 
to grow, how are campuses sharing this information with the wider world? To what degree have colleges 
and universities been willing to report publicly and share what they are doing and learning? 


‘These are the questions to which this set of NILOA reports speaks. The work represents a series of 
studies that NILOA staff, led by Staci Provezis and Natasha Jankowski conducted over the past couple 
of years, and may be the most comprehensive examination yet of institutional transparency in terms of 
student learning outcomes and the assessment activities. 
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Foreword continued 


‘The study is based on a well-designed web scan approach whereby the publicly accessible areas of 
college and university websites were scoured for mentions of student learning outcomes assessment 
and the evidence such efforts yielded along with examples of how the data were used. One of the more 
informative findings is that on our national surveys campuses report more assessment activity going on 
than Staci, Natasha, and their colleagues could find on the website. 


These reports can be read as one package or separately as time and interest permit. What you 

will find from these analyses, the Featured Website page on the NILOA website (see http:// 
www.learningoutcomesassessment.org/featured-websites), and the examples of institutions 

using the NILOA Transparency Framework (see https://www.learningoutcomesassessment.org/ 
ourwork/transparency-framework/) is that some colleges and universities are well out in front of the 
pack in terms of sharing information on student learning. Still, higher education has a distance to go to 
adequately communicate to the several interested parties the myriad activities underway at the class, 
program, and institutional levels. Moreover, we have a lot to learn about how to make this wealth of 
information understandable and meaningful to students, parents and policy makers. Learning what 
students know and can do and using that evidence to make wiser decisions and improve student 
success remain huge challenges. 


Finally, taken together, these reports establish the baseline for future web scan studies. That is, these 
ground-breaking efforts will allow us to estimate the progress we will make down the line in addressing 
the transparency issue. And for that, we are grateful to the authors and the research team for their 
excellent work and insights. 


George Kuh 

Director, National Institute for Learning Outcomes Assessment 
Adjunct Professor, University of Illinois 

Indiana University Chancellor’s Professor Emeritus 


Stan Ikenberry 


President Emeritus and Regent Professor 
University of Illinois 
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Section l 


Revealing Student Learning 
Outcomes: National Transparency 
Initiatives Make a Difference 


Clarion calls nationwide exhort institutions of higher education to increase postsec- 
ondary credential and degree attainment while simultaneously increasing academic 
quality. Regional and special field accrediting groups also hold colleges accountable 
for the assessment of student learning and the use of that evidence for improvement. 


Some argue that making assessment information more widely available and readily iii Sy 


accessible—more transparent—will enable prospective students and their families 

to make better informed college-going decisions and will also aid institutions in 

deploying their scarce resources. This report examines the impact 
ar of these national transparency 

No medium offers more possibilities for transparency than the Internet. Institu- initiatives and efforts by 

tion web pages, for example, are accessible to interested parties on and off campus. agri 

Utilizing the Internet's potential for enabling transparency, several national higher institutions to make student 

education organizations have created online templates with which member institu- learning outcomes information 

tions can publicly present information about their students and their institution’s available on their websites. 

performance. This report examines the impact of these national transparency initia- 

tives and efforts by institutions to make student learning outcomes information avail- 

able on their websites. 


Four initiatives were the focus of this investigation: the University and College Account- 
ability Network (U-CAN), the Voluntary System of Accountability (VSA), Transpar- 
ency By Design (TbD), and Achieving the Dream (ATD).' This study is part of an 
ongoing series of efforts undertaken by the National Institute for Learning Outcomes 
Assessment (NILOA).? The findings from this analysis suggest the following: 


1. The majority of TbD and VSA institutions posted student learning outcomes 
assessment results. For the most part, institutions participating in U-CAN 


and ATD did not do so. 


2. Institutions participating in VSA posted more examples of actual use of 
learning outcomes evidence than did institutions in other national transpar- 
ency initiatives. 


3. Institutions participating in national transparency initiatives tended to make 
more assessment information public than did nonparticipating institutions. 


4, Public institutions showed more assessment information than did indepen- 
dent institutions, even those participating in a national transparency initia- 
tive. 


5. Whether participating in a national transparency initiative or not, institu- 
tions that had been accredited between 2008 and 2010 were also more likely 
to make student learning information available. 


' For full descriptions of each of these initiatives, see Table 1. 
> See Exploring the Landscape: What Institutional Websites Reveal about Student Learning Outcomes Assess- 
ment Activities, retrievable from https://www.learningoutcomesassessment.org/wp-content/ 


uploads/2019/02/WebScanReport.pdf 
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The NILOA 2010 Web Study 


The NILOA 2010 Web Study sought to document the relationship between 
institutions participating in national transparency initiatives and disclosure 
of assessment information on institution websites. A team of three NILOA 
researchers examined 200 institution websites between July 2010 and 
September 2010 to answer the following questions: 


1. What and how much information do institutions display on their 
website regarding student learning outcomes assessment? 


2. To what extent do institutions show results and describe their use of 
the results on their institution web pages? 


3. Is there a difference in learning outcomes disclosure on websites 
between institutions involved in national transparency initiatives and 
institutions not involved in those initiatives? 


The sample for this web study was composed of two sets of institutions: 100 
institutions participating in national transparency initiatives and 100 compa- 
rable institutions not participating in national transparency initiatives. Institu- 
tions in both groups were similar in size, control, setting, type, and regional 
accreditation.? The national transparency initiatives included the Univer- 
sity and College Accountability Network (U-CAN), the Voluntary System 
of Accountability (VSA), and Transparency by Design (TbD). Although 
Achieving the Dream (ATD) is not a transparency initiative per se, it was 
included in this study to get a sense of assessment activity within the two-year 
sector given that the Voluntary Framework of Accountability (VFA) had not 
started its pilot effort until four months after the research for this study was 
completed. However, ATD does focus on using data to improve practices and 
student success through institutional change, creating a culture of evidence, 
and collecting data and publicizing findings on student success.‘ 


Some relevant definitions used in this report include: 


Results. To show results means to provide indicators of institutional perfor- 
mance as well as indirect and direct evidence of student learning. 


¢ Performance indicator. An indicator of overall student performance, 
which may include licensure pass rates, graduation rates, placement 
rates, common data sets, fact books, and so on. 


¢ Direct evidence of learning. An indicator of what students have 
learned in a concrete way. Direct evidence may include results of 
capstone experiences, portfolios, e-portfolios, standardized tests, and 
local tests. 


¢ Indirect evidence of learning. An indicator that represents or approx- 
imates what students have learned, without providing a concrete 
demonstration of that learning. Indirect evidence may include results 
of national student surveys, local student surveys, graduating student 
surveys, alumni surveys, and other similar instruments. 


National Transparency Initiatives 


Each of the voluntary transparency initiatives asks its member institutions to 
use a common web-based reporting template to publicly display information. 
Table 1 summarizes the basic features of each transparency initiative. 


3 For additional information on the study design and sampling, see Appendix A. 
“ Of the ATD institutions examined in this study, ten were selected to participate as pilot colleges 
for the VFA and an additional eight were part of the study. 


Each of the voluntary 
transparency initiatives asks 

its member institutions to use 
a common web-based reporting 
template to publicly display 


information. 
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Table 1. Overview of National Transparency Initiatives 


University and College Account- 
ability Network (U- ) 


http://www.ucan-network.org/ 


Web-based resource designed to give 
prospective/current students and their fami- 
lies concise, consumer-friendly information 
on nonprofit, independent colleges and 
universities in a common reporting format. 


Voluntary System of Account- 
ability (VSA) 
http://www.voluntarysystem.org/index. 
cfm 


Web-based resource designed to provide 
information to students, families, policy 
makers, campus faculty and staff, the 
general public, and other higher education 
stakeholders on the undergraduate student 
experience through a common web- 
reporting template, the College Portrait. 
http://www.collegeportraits.org/ 


Transparency By Design (TbD) 
http://wcet.wiche.edu/advance/ 
transparency-by-design 


Web-based initiative to lead universities and 
colleges toward greater accountability and 
transparency. College Choice for Adults is 
an online reporting template launched in 
2009 to help users determine what they will 
learn from each program. 
http://www.collegechoicesforadults.org/ 


Achieving the Dream (ATD) 


http://www.achievingthedream.org/ 


A national initiative for student success 
through encouraging institutions to use 
data to guide changes in institutional 
policy, programs, and practices to promote 
educational attainment, particularly among 
low-income students and students of color. 


Voluntary Framework of 
Accountability (VFA) 
http://www.aacc.nche.edu/Resources/ 
aaccprograms/VFAWeb/default.aspx 


A system for community colleges to 
benchmark student information with peers 
around issues of completion and success as 
well as gather and use data to improve their 
effectiveness. 


Developed in September 2007 and 
managed by the National Associa- 
tion of Independent Colleges and 
Universities (NAICU) for indepen- 
dent, not-for-profit institutions. 
Over 800 institutions participate. 


Developed in 2007 by the Amer- 
ican Association of State College 
and Universities (AASCU) and the 
Association of Public and Land- 
Grant Universities (APLU) for four- 
year public colleges and universities. 
Over 300 institutions participate. 


Developed in 2004 by the Presi- 
dent’s Forum, a group designed to 
provide a means for leaders to learn 
from one another in a collaboration 
of accredited, adult-serving institu- 
tions to advance innovation and 
excellence in online learning. For 
more information, see http://presi- 
dents forum.excelsior.edu/. TbD is 
open to all regionally-accredited, 
adult-serving higher education 
institutions that offer some or all 
programs at a distance. Roughly 18 
institutions participate. 


Developed in 2004 through a 
Lumina grant and seven founding 
partner organizations it focuses 
on community colleges. Over 100 
institutions participate. 


Being developed by the American 
Association of Community Colleges 
(AACC), the College Board, and 
the Association of Community 
College Trustees (ACCT) to serve 
community colleges and other two- 
year institutions. 


Information in the 
institution profiles 
includes admissions, 
enrollment, academics, 
and other performance 
indicators. 


Information in the 
College Portrait 
includes consumer 
information, student 
experiences and percep- 
tions (generally shown 
through posting NSSE 
results) and student 
learning outcomes 
(generally shown 
through posting results 
of a standardized test). 


Information provided 
includes program 
specific learning 
outcomes data, nation- 
ally comparable infor- 
mation on student 
engagement (through 
nationally normed 
surveys) and alumni 
satisfaction (through a 
common set of ques- 
tions). 


Emphasizing a data- 
driven approach, the 
initiative works on 
three main fronts: 
institutional (through 
coaching and campus 
culture), knowledge 
(collecting institutional 
data), and policy. 


The information that 
will be included on the 
VFA is under develop- 
ment with the intention 
to create metrics that 
will serve the variety of 
students and purposes 
of the American 
community college 
system. 


‘The U-CAN template does 

not include learning outcomes 
measures because in their focus 
groups of students and parents 
on information needed to inform 
college choice there was no 
consumer demand found for the 
information (National Associa- 
tion of Independent Colleges and 
Universities, 2010). 


The VSA was formed out of a task 
force of university presidents and 
other administrators who met to 
discuss what it is that the public 
wants to know about choosing 
colleges. 


College Choice for Adults includes 
program-level learning outcomes 
and program-level results to 
assist adult students in selecting 
a program consistent with their 
personal and professional goals. 
Users of College Choice for 
Adults may compare institu- 
tions by selecting several they are 
interested in learning more about 
and viewing the relevant learning 
outcomes. 


At this time, ATD does not 
directly address student learning 
outcomes assessment beyond 
focusing on performance indi- 
cators, but they do encourage 
institutions to benchmark and 
analyze student outcomes, employ 
data-driven decision making, and 
provide better access to data, poli- 
cies, and practices that improve 
student outcomes. 


Forty institutions began pilot 
testing the VFA in February 2010 
and full availability to all commu- 
nity colleges will start in 2012. 
This initiative was not part of the 
study as it is still in development. 
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Findings 


National Transparency Initiatives Websites 

Initiative-specific websites were reviewed to determine how many of the 100 institu- 
tions participating in the national transparency initiatives posted learning results on 
the respective common reporting template (Figure 1). 


100% 
90% 
80% 
& 70% 
2 
3 60% 
3 50% 
D 
£ 40% 
8 
5 30% 
20% 
10% 
0% 7 . 
TbD VSA ATD U-CAN 
National Transparency Initiatives 


Figure 1. Percentage of Institutions Showing Results of Student Learning on the Website 
of Their National Transparency Initiative 


¢ Transparency by Design institutions (93%) were the most likely to share 
learning results on the College Choices for Adults website, followed by VSA 
institutions (87%) on the College Portrait site. Results included selected 
survey findings from alumni surveys and national student surveys such as 
National Survey of Student Engagement (NSSE) as well as results from more 
direct assessment of learning such as the Collegiate Learning Assessment 


(CLA). 


¢ U-CAN does not call on institutions to make student learning information 
transparent, and the U-CAN template does not provide a specific section for 
such information. Institutions that participated in U-CAN did not display 
student learning outcomes results on their U-CAN pages; however, they did 
post information on standard institutional performance indicators such as 
enrollment numbers, graduation rates, and cost. 


¢ Achieving the Dream institutions did not have a template for institutions 
to use to share results of student learning; however, 7% of the institutions 
posted results of student attainment on the ATD initiative website. 


More specifically, 


¢  TbD institutions: 100% posted performance indicators, and 86% posted 
selected NSSE results and alumni survey results. 


¢ U-CAN institutions: 89% posted performance indicators, but none posted 
results of assessment activities; however, 11% did provide links to assessment 
information via their institution website. 


e VSA institutions: 13% posted performance indicators, 84% posted selected 
NSSE results, 26% posted CLA results, and 16% posted ETS proficiency 
profile results. Of the institutions using College Portraits, 55% provided 
links to assessment information on their institution website. 


e ATD institutions: 67% posted performance indicators, and 30% stated 
that a specific goal of the project was to use data to improve practices and 
outcomes. 


Transparency by Design 
institutions (93%) were the 

most likely to share learning 
results on the College Choices for 
Adults website, followed by VSA 
institutions (87%) on the College 
Portrait site. 
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Institutions Participating in National Transparency 
Initiatives 
‘The websites of the 100 institutions participating in national transparency initiatives 


were also 


examined to determine the extent to which they posted assessment informa- 


tion and advertised their participation in a national transparency initiative. As Figures 
2 and 3 show, institutions varied in terms of whether they publicly acknowledged 
participation in national transparency initiatives or how much assessment information 


they disclosed. 
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National Transparency Initiatives and All Institutions 


Figure 2. Percentage of Institutions Posting Their Participation in a National Transparency 


Initiative on Their Institution Website 


Of the 100 institutions participating in national transparency initiatives, 
only a third (32%) listed their participation on their institution website. 


About three quarters (77%) of VSA institutions mentioned their participa- 
tion, with the majority posting the information on the institutional research 
or assessment page. 


Only 9% of VSA institutions in the sample indicated on their home or 
admissions page that they were participating in VSA.° 


21% of TbD institutions posted their participation in College Choice for 
Adults on their home and/or admissions page. 


20% of ATD institutions posted their participation on their home or insti- 
tutional research web pages. 


U-CAN institutions were the least likely to post their participation on their 
institution website, as well as the least likely to post information from surveys, 
tests, and results on their institution websites than institutions participating 
in other initiatives. 


More VSA institutions posted information on their institution websites from 
surveys, portfolios, standardized tests, faculty/staff development, awards for 
assessment, results, and examples of use than institutions participating in 
other initiatives. 


Fewer TbD institutions posted information about faculty/staff development, 
local student surveys, and performance indicators than institutions partici- 
pating in other initiatives. 


> This finding shows a shift from a NILOA study completed last year. Of the VSA institutions scanned 
last year, 75% of the institutions that previously posted their participation on their home page no 
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longer d. 


Of the 100 institutions 
participating in national 
transparency initiatives, only 

a third (32%) listed their 
participation on their institution 
website. 
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Figure 3. Percentage of Institutions Participating in a National Transparency Initiative Posting Assessment Information on Their 
Institution Website 


Institutional Control 

Because these national transparency initiatives were developed with their institution members in mind, some of the differences 
between them are more a function of the differences in the characteristics of the member institutions than of the initiatives 
themselves. This is very evident in Figure 4, which compares public and independent institutions. 
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Figure 4. Percentage of Institutions Participating in a National Transparency Initiative Posting Assessment 
Information on Their Institution Website by Institutional Control 
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¢ Fewer independent institutions than public institutions posted information 
on all assessment activities including results of assessment and their use. The 
only exception was in respect to capstone experiences, on which independent 
institutions posted more information than did public institutions. 


Institutional Comparison 
Institutions participating in the national transparency initiatives were matched with 


institutions that did not participate in such initiatives but were similar in size, setting, Because these national 
control, and regional accreditation. The sample of institutions participating in U-CAN, for transparency initiatives were 
instance, was matched with a sample of similar institutions not participating in U-CAN to develop ed with their institution 


determine if any differences were found. ; 9 
- members in mind, some of the 


differences between them are more 
a function of the differences in 

the characteristics of the member 
institutions than of the initiatives 
themselves. 


¢ There was no difference in terms of the nature or amount of assessment 
information on the institution website between institutions that were partic- 
ipating in U-CAN or ATD and those that were not. 


* More VSA institutions posted information from standardized tests, national 
student surveys, and other learning results than did non-VSA participating 
institutions with similar characteristics. 


¢ More TbD institutions showed information from national student surveys, 
alumni surveys, and results on their institution website than did non-TbD 
institutions. 


The findings for VSA and TbD institutions were expected, given that the templates of 
these initiatives specifically required information from surveys and standardized tests. 
In this sense, these national transparency initiatives appeared to influence the kind 
of information institutions make publicly available. Conversely, because the U-CAN 
template did not require such information, participating institutions, not surprisingly, 
were far less likely to make it available. 


The Impact of the National Transparency Initiatives 
In this section, we summarize the major findings across the 100 institutions partici- 
pating in national transparency initiatives and the 100 institutions not participating 
in them. 
¢ — Institutions involved in national transparency initiatives were slightly more 
likely than those not involved in initiatives to post on the institution website 
results of student learning outcomes assessment, examples of the use of those 
results, and performance indicators (Figure 5). 
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Figure 5. Percentage of Institutions Posting Assessment Information by Participation in Initiative 
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¢ — Institutions participating in national transparency initiatives were more likely 
to report results from national student surveys (70%) than were those not 
participating (44%). 


¢ Those participating in national transparency initiatives also were more likely 
to post information from standardized tests (30%) than were those not 


participating (17%). ee 


In most cases, accreditation region 
80% was not related to the posting of 
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Figure 6. Percentage of Institutions Showing Assessment Activity by Participation in Initiative 


Regional Accreditation 

The expectations of regional accreditors may influence the transparency of learning 
outcomes (Jankowski & Makela, 2010; Kuh & Ikenberry, 2009; Provezis, 2010). 
To get a sense of whether posting assessment information was a function of recent 
accreditation activity (accreditation team visits within the past two years), information 
regarding institutional accreditation was examined to see whether a recent accredita- 
tion visit may have influenced the information found on the institution website. Four 
findings warrant mention: 


e Institutions accredited within the past two years, meaning between 2008 
and 2010 were more likely to post student learning outcomes statements and 
examples of the use of evidence of student learning than were those not 
accredited within the past two years. 


¢ — Institutions accredited within the past two years were more likely than those 
not accredited recently to post information on standardized tests, alumni 
surveys, and portfolios. 


¢ Institutions accredited within the past three years were more likely than those 
not accredited between 2007 and 2010 to provide examples of the use of 
assessment results on their websites. 


¢ — Institutions preparing for accreditation in the upcoming years, specifically 
2011 and 2012, did not display any significant differences in the information 
posted on their websites. 


In most cases, accreditation region was not related to the posting of assessment activity 
information on institution websites. However, institutions were Jess likely to publicly 
post assessment results on their website if they were members of the Middle States 
Association of Colleges and Schools and the New England Association of Schools and 
Colleges than if they were members of the other regional accreditation organizations. 
This may be influenced in part, however, to the larger proportion of independent 
institutions in the Middle States and New England regions. 
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Conclusions 


Overall, findings of this study point to five conclusions: 


1. Institutions participating in TbD and VSA tended to post institutional 
results of assessment on their initiative websites to a greater extent than did 
institutions participating in other national transparency initiatives. 


2. Institutions participating in VSA posted more assessment information, 
results, and examples of use on their institution websites than did institu- 
tions participating in other national transparency initiatives. 


3. For all sectors, however, those institutions participating in national transpar- 
ency initiatives tended to make transparent on their website more informa- 
tion about student learning outcomes and display more examples of how 
these results were being used. 


4, Recently accredited institutions were more likely to post examples of use of 
student learning outcomes evidence. 


5. With the exception of information on capstone learning experiences, inde- 
pendent institutions were less likely than public institutions to post student 
learning outcomes assessment information. 


Participation in national transparency initiatives appears to encourage more transpar- 
ency in reporting of results and of the use of assessment information. Accreditation 
makes a difference as well, especially when it comes to demonstrating the use of assess- 
ment results. It is self-evident that online reporting template designs and requirements 
have a substantial effect on the information that gets reported and—further upstream, 
perhaps—on the kind of information the institution collects and uses to make deci- 
sions. Independent institutions are not expected via the U-CAN template to report 
student learning outcomes information. At the same time, it is clear by the numbers 
of independent colleges participating in initiatives such as the Teagle consortia and the 
Council of Independent College-coordinated efforts to use the CLA and NSSE, that 
learning outcomes assessment activity is underway. In short, while participation in 
national transparency initiatives may increase transparency by inducing campuses to 
make assessment information publicly available, websites and other public venues are 
not the only indicators of an inducement toward enhanced transparency. 


While participation in national 
transparency initiatives may 
increase transparency by inducing 
campuses to make assessment 
information publicly available, 
websites and other public venues 
are not the only indicators of an 
inducement toward enhanced 
transparency. 
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Section 2 


The Changing Landscape of 


Transparency in Student Learning 


Outcomes Assessment 


Assessing student learning and using that information wisely is an important func- 
tion of higher education. Communicating these efforts and making them transparent 
presents complex challenges to higher education institutions. Among these challenges 
is the reality that, for the communication of their assessment information, institutions 
have two very different audiences with very different interests, purposes, and perspec- 
tives: external audiences and internal audiences (Ewell, 2009; Suskie, 2009; Volk- 
wein, 2008; Walvoord, 2004; 2010). External audiences, such as prospective students, 
families, policy makers, and tax payers, are interested in knowing about student 
learning. Assessment information targeting these audiences would highlight the insti- 
tution’s successes and disclose what the institution is doing to address any shortcom- 
ings (McCormick, 2010). Assessment information targeting internal audiences, on 
the other hand, audiences such as faculty members, academic leaders, and governing 
boards, would highlight information on student performance and the possibilities to 
improve teaching and learning. While these external and internal audiences are often 
considered separate, the distinctions between them are fluid and often unclear. To 
effectively communicate information about student learning outcomes and institu- 
tional performance, institutions must develop ways to respond to a complex multitude 
of needs and interests. 


Over the last three years, the National Institute for Learning Outcomes Assessment 
(NILOA) has been investigating the state of student learning outcomes assessment in 
USS. higher education. These studies have included a survey of chief academic officers 
and a second survey of departments regarding assessment practices. NILOA has also 
conducted a number of focus groups, reviewed institution websites, examined state 
policies, and conducted institution case studies in an attempt to learn more. Looking 
across these investigations, what have we learned about how the assessment landscape 
has changed—particularly the landscape of institutional transparency of student 
learning outcomes assessment? To answer this question, this report synthesizes key 
findings from three studies: the NILOA National Survey of Chief Academic Officers, 
conducted in 2009; the NILOA 2009 Web Study; and the NILOA 2010 Web Study. 


Studying the Role of Institution Websites in Transparency 


While many forms of media can facilitate institutional transparency, none is more 
valuable and amenable to the task than the Internet and the institution web page. 
To discern what information about student learning was accessible and where that 
information was most likely to be found online, NILOA systematically examined 
institution web pages. The first NILOA web study was done in 2009 and a second in 
2010. This report examines the shift in transparency patterns during this brief one- 
year period. Our findings from this analysis confirm the following: 


1. Higher education institutions are showing more information online now 
than previously about student learning outcomes assessment. 

2. While more information on student learning outcomes assessment is being 
made available, likely only a fraction of assessment activity, as reported in the 
2009 NILOA National Survey of Chief Academic Officers (Kuh & Iken- 
berry, 2009), is being made transparent. 


While many forms of media 
can facilitate institutional 
transparency, none is more 
valuable and amenable to the 
task than the Internet and the 
institution web page. 
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The NILOA National Survey of Chief Academic Officers 


In spring 2009, NILOA invited chief academic officers at all regionally accredited, 
undergraduate-degree-granting, two- and four-year, public, private, and for-profit 
institutions in the U.S. (n=2,809) to respond to a series of questions about the 
assessment activities underway at their institutions and how assessment results are 
being used. All told, 1,518 institutions responded, or 53% of the original sample. 
The NILOA National Survey of Chief Academic Officers instrument was organized 
around four broad questions that addressed what learning outcomes institutions were 
measuring, how they were assessing the outcomes and using the results, and the factors 
prompting assessment at the institution.! 


The NILOA 2009 Web Study 


In its 2009 Web Study, NILOA sought to document what institutions portrayed in regard 
to student learning outcomes assessment on institution-wide web pages (as opposed to 
department or program-based web pages) by examining the assessment activities, the web 
page locations, and the language that institutions used on their websites regarding student 
learning outcomes assessment. Institutions were randomly selected from a group of all 
regionally accredited undergraduate degree-granting postsecondary institutions in the U.S. 
A team of six researchers systematically reviewed 725 institution websites from March 
2009 to August 2009. In contrast to the 2010 Web Study, no attempt was made in the 
2009 Web Study to determine whether the institution reported the results of its assessment 
efforts.’ 


The NILOA 2010 Web Study 


To conduct the 2010 Web Study, a team of NILOA researchers spent roughly 120 
hours reviewing 200 institution websites from July 2010 to September 2010, seeking 
answers to the following questions: 


1. What and how much information do institutions display on their website 
regarding student learning outcomes assessment? 


2. Which institution web pages provide information about assessment of 
student learning outcomes? 


3. To what extent do institutions show results and describe their use of the 
results on their institution web pages? 


The 200 institution websites scanned were examined for information on assessment 
activities, student learning outcomes statements, posted results, and examples of the 
use of assessment findings. Web pages examined for information on student learning 
outcomes assessment included home, admissions, institutional research, provost/ 
academic affairs, center for teaching and learning, regional accreditation, and assess- 
ment pages. For more information on the methodology and sample, please see the 


Appendices. 


Findings 


1) Institutions are showing more information online now than previ- 
ously about student learning outcomes assessment. 


‘For more information on the survey instrument, see 2010 Survey Questionnaire, retrievable from https:// 


www.learningoutcomesassessment.org/wp-content/uploads/2019/08/NILOA-2010-Program- 
Survey.pdf. For findings from the survey, see More Than You Think: Less Than We Need, retrievable 
from https://www.learningoutcomesassessment.org/wp-content/ 
uploads/2019/02/2009NILOASurveyReport.pdf 


For more information on the 2009 Web Study including the methodology and findings, see: Exploring 
the Landscape: What Institutional Websites Reveal About Student Learning Outcomes Assessment 
Activities, retrievable from https://www.learningoutcomesassessment.org/wp-content/ 


uploads/2019/02/WebScanReport.pdf 


Of the institutions examined 

in the 2010 Web Study, 49% 
created web pages solely devoted 
to providing information on 
student learning outcomes 
assessment. 
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The 2010 Web Study found institutions were participating in multiple aspects of 
assessment activity through listing student learning outcome statements and providing 
information on assessment activities on their websites. For instance, 49% of the institu- 
tions examined in the 2010 Web Study created web pages solely devoted to providing 
information on the assessment of student learning and focused on communicating 
learning outcomes information to multiple audiences. In addition, information on 
the assessment of student learning was found on each of the diverse web pages exam- 
ined; slightly over half (52%) of the institutions scanned had institution-wide student 
learning outcomes statements posted on their website. Of those that had statements, 
61% were easily found on the website while 39% were difficult to locate or hidden 
within the text of larger documents. Comparing the results of the 2010 Web Study 
with those of the 2009 Web Study (Figure 2.1), we found the following: 
¢ In the 2009 Web Study as well as in the 2010 Web Study, the most commonly 
found type of assessment-related information posted on institution websites 
was information from national student surveys, followed by information 
from local student surveys, alumni surveys, and other surveys. Information 
from local tests was found the least often on institution websites in both web 
studies. 


¢ The 2010 Web Study found more information with indirect evidence of 
student learning than was found in 2009 but less evidence of capstones and 
portfolios. 
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Figure 2.1. Percentage of Institutions Posting Information on Assessment Activity, Results of 2009 and 2010 Web Studies 


While more information was found in the 2010 Web Study, researchers also noted 
that institutions were password-protecting assessment information more often than 
found in the 2009 Web Study. While 9% of the institution websites in the 2009 Web 
Study had password-protected. assessment-related information, this percentage had 
increased to 15% in the 2010 Web Study. This growth suggests that to house assess- 
ment results institutions may be using internal management software applications 
such as WEAVE online or TracDat, which require a login and limit external audience 
access to information. 
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2) While more information on student learning outcomes assess- 
ment is being made available, institutions still fall short of 
reporting as much assessment activity as chief academic officers 
have indicated. 


In comparing findings from the 2009 Web Study with those from the NILOA 
National Survey of Chief Academic Officers, researchers found that institutions had 
claimed in their survey responses to be involved in more assessment activities than 
were represented on their institution websites. Because institutions in the 2010 Web 
Study were found to be posting more information on their assessment activities on 
their websites than in the 2009 Web Study, the findings across all three studies were 
compared to examine change over time (Figure 2.2). 


nn 
c 
° 
= 
5 
2 
= 
”n 
c 
uw 
° 
- 
c 
o 
2 
o 
oa 


National Student —_ Local Student Survey Alumni Survey Other Survey Standardized Tests 
Survey 


@ National Survey 2009 Web Study #2010 Web Study 


Figure 2.2. Percentage of Institutions Displaying Information on Assessment Activities, Results of National Survey and 2009 and 
2010 Web Studies 


¢ Institutions currently show significantly more assessment activities on their 
websites than before, but still not as much as suggested in the National 
Survey results. 


¢ The gaps between stated assessment activity and information found online in 
2010 are not nearly as large as those found in 2009. 


¢ Information on local student surveys and other surveys was found online 
more often in 2010 than was suggested in the National Survey results. 
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Ways to Enhance Communication 


Institutional transparency related to student learning outcomes information appears 
to be increasing, even in this report’s brief time span of one year. Institutions have 
begun to share more assessment information on their websites, but they may still not 
be showing as much outcomes assessment information as they have available. While 
student learning outcomes statements were found at 52% of the institutions, these 
statements were often difficult to find. Even so, according to the NILOA National 
Survey, some three quarters of institutions claim to have statements of goals and 
expectations for student learning. These findings illustrate how increasing transpar- 
ency could aid communications with institutions’ external as well as internal audi- 
ences. Following are specific suggestions to enhance communication about student 
learning: 


1. Gather assessment information, such as statements of learning goals and 
expectations assessment measures, and reports, considering what informa- 
tion would be of interest to internal and external audiences. 


2. Think about how best to communicate with each audience. A detailed report 
may work for an internal audience; but graphs and simple explanations that 
avoid jargon may serve an external audience best. 


3. If information is password protected, consider what portions could be 
extrapolated and communicated to external audiences. 


4. Consider where assessment information should be displayed and how users 
can navigate to the page. Our study found that 49% of institutions place 
information on a single page and direct users throughout the website to that 
page. 

5. Check the NILOA Transparency Framework for ways to make each compo- 
nent of your assessment activity more transparent.? 


6. Finally, to avoid embarrassment, take time to review your website regularly 
to assure that all information is up-to-date. 


> See https://www.learningoutcomesassessment.org/ourwork/transparency-framework/ 
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Section 3 


Outcomes Assessment Results 


The National Institute for Learning Outcomes Assessment (NILOA) reviews assess- 
ment activity at colleges and universities and offers resources to campuses to assist and 
strengthen assessment initiatives. Some of NILOA’s efforts have centered on institu- 
tional transparency with assessment information. Internet websites serve as a primary 
medium of communication for most colleges and universities. How institutions 
convey information about student learning on their websites is of major interest to 
institutions, accrediting groups, and other higher education stakeholders. In an initial 
study, conducted in 2009, NILOA staff examined the websites of over 700 colleges 
and universities to determine whether and where student learning outcomes assess- 
ment was mentioned (Jankowski & Makela, 2010). In a follow-up study, conducted 
in 2010, NILOA staff looked specifically at whether institutions share information on 
assessment results and how those resu/ts were actually used. In this second study some 
200 college and university websites were examined over a three-month period. This 
report of that study describes findings that add to our understanding of the broader 
issue of transparency. The results of the study suggest four conclusions: 


1. Most institutions post some results of student learning outcomes assessment, 
but few provide examples of the use of those results on their institution 
websites. 


2. Often, the posted results are not presented in user-friendly ways. 


3. Institutions use assessment results in many ways, the most prominent being 
for accountability purposes and for institutional improvement. 


4, Regional accreditation self-study is the most cited use for assessment results. 


The NILOA 2010 Web Study 


This NILOA web study examined institution websites for assessment activity, public 
disclosure of results, and examples of the use of assessment findings. A team of NILOA 
researchers systematically reviewed 200 institution websites over 120 hours from July 
2010 to September 2010 to address the following guiding questions (see Appendix A 
for further information on the study design and sampling): 


1. What and how much information do institutions display on their websites 
regarding student learning outcomes assessment? 


2. Which institution web pages provide information about assessment of 
student learning outcomes? 


3. ‘To what extent do institutions show results and describe their use of the 
results on their institution web pages? 


Findings 


The 200 institution websites were scanned for information on posted results of student 
learning outcomes assessment and examples of the use of assessment findings. This 
report provides an overview of those findings in three sections including a general 
overview, an in-depth section looking at cases in which results were publicly shared 
online, and a section examining posted examples of the use of assessment findings. 
Several vignettes showing specific examples of institutional activity online are also 
provided throughout the report. 


Presenting and Using Student Learning 


How institutions convey 
information about student 
learning on their websites is of 
major interest to institutions, 
accrediting groups, and other 
higher education stakeholders. 
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Overview 


The following percentages show general findings on the 200 institution websites 


reviewed in this study (Figure 3.1): 


* 78% posted performance indicators; 
* 57% posted results of student learning outcomes assessment; and 


* 34% posted examples of the use of student learning outcomes assessment 


results. 
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Figure 3.1. Percentage of Institutions Posting Performance Indicators, Assessment Results, 


and Examples of the Use of Assessment Results on Their Institution Websites 


Performance Indicators 


Examples of performance indicators found on institution websites include student 
demographics, cost, enrollment data, fact books and common data sets, graduation 
rates, pass rates on licensure exams, and retention rates. Performance indicators were 
often presented in a list, in a report, or in pie graphs. Occasionally, performance 
indicators were disaggregated by race/ethnicity, gender, or full-time/part-time status. 
Performance indicators do not directly relate to student learning, but they may be used 
in conjunction with the results of student learning outcomes assessment to present a 
more complete picture of student performance or to help decision makers identify 
areas for improvement. Findings from the 200 institution websites on performance 


indicators include the following: 


¢ Institutions were less likely to post the results of student learning outcomes 
assessment than they were to post information on performance indicators. 
Still, over half (57%) of the institutions in the study publicly posted the 
results of student learning outcomes assessment activities on their institution 


websites. 


¢ — Institutions were much less likely to post examples of the use of the results of 
assessment (34%). Even in the instances where institutions clearly outlined 
their use of student learning outcomes assessment results on their website, 
not one institution in the study included information on whether the changes 
informed by assessment results actually led to improvements in outcomes or 


in student learning. 


Performance indicators found on institution websites appeared as “consumer informa- 


tion,” such as on Augustana College’s Open Book Program (see vignette 1). 
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Assessment Results 

Results of student learning outcomes assessment should be presented in ways that 

facilitate use by students, parents, faculty members, and institution decision makers— 

such as being displayed in tables as well as in succinct, jargon-free text tailored to 

specific audiences. Information should also be provided on assessment measures and Sy — 
on recommended or implemented changes (Ketcheson, 2001; Middaugh, 2010; 

Suskie, 2009; Volkwein, 2010; Walvoord 2004; 2010). 


An example of an attempt to make assessment findings user friendly is found on 


Capella University’s web page showing results of student learning (see vignette 2). Vignette I: 

Posted results at the 200 institution websites were mainly composed of surveys, specif- Augus. tana College P rovides a 
ically national student surveys, and standardized test results (Figure 3.2). This is not web page tool under their general 
surprising since the majority of information on assessment activity is survey based information tab to help multiple 


(Jankowski & Makela, 2010). Of institutions that used student portfolios as part of 


aa) aaron audiences assess their college 
their assessment activities, no aggregate results or results on individual student portfo- 


lios were found. Rather, viewers could access individual portfolios for samples of work options. The page, titled the 
or examine rubric templates used to assess the work. Open Book Program, includes 
100% links to performance indicators; 


governmental data; selected 
first-year, senior, and NSSE 

80% survey results; as well as alumni 
outcomes information. The 

data are not linked collectively 
but are provided in a central 
location and tailored to external 
audiences, thus presenting 
performance indicators alongside 


on data on student learning. 
20% http://www.augustana.edu/ 
‘an: | | x3148.xml 
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Figure 3.3. Percentage of Institution Web Pages Displaying Results of Student Learning 
Outcomes Assessment 
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Results of student learning outcomes assessment were located most often on the insti- 
tutional research web page, with 61% posting publicly available results (Figure 3.3). 


Assessment web pages showed results 54% of the time, followed by accredi- 
tation web pages at 36%. 


The web pages least likely to show results of student learning were the center 
for teaching and learning and admissions pages. 


In regard to the actual reporting of assessment results, institutions posted results in 
multiple formats and in many different ways (Figure 3.4). 


The majority of institutions (67%) posted reports that included complete 
results of surveys or standardized tests as opposed to selected, highlighted, or 
partial results (33%). Further, 54% of institutions posted multiple reports of 
different assessment activities such as surveys or tests on their website. 


The majority of reports (54%) included long tables of data that were hard 
to read or comprehend. Graphs of results were provided in 46% of the insti- 
tution reports, with only 29% of institutions providing text-based reports 
without figures, graphs, or tables. 


In the posted reports, 70% of the institutions did not benchmark the results 
with peer institutions or previous results, and 22% of the reports were over 
100 pages in length. 


Few institutions presented a summary of key findings at the beginning of the 
report (30%), while even fewer discussed background information on what 
the survey/instrument measured (16%), provided implications or next steps 
(10%), or tailored their reports to a specific audience (10%). 


Few institutions (21%) included longitudinal data where results were 
compared over time, while even fewer disaggregated results (10%) or 
combined results of multiple assessments (3%). 


Few institutions provided PowerPoint presentations of their results (10%), 
while still fewer advertised or marketed their results to a broader audience 


(5%). 


Information on Sampling or Survey 
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Vignette 2: 


Capella University has a website 
devoted solely to the reporting of 
results of student learning. The 
site includes interactive charts 
and graphs by program as well as 
career outcomes, alumni survey 
responses, and explanations of 
how student learning is measured 
and reported—all in one 
centralized location. http://www. 


capellaresults.com/index.asp 


Figure 3.4. Percentage of Institutions Posting Assessment Results by Selected Reporting Charac- 


teristics 
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Some institutions provide assessment data in interesting ways to viewers, such as 
through a changing Fact of the Day at Case Western Reserve or through an Interesting 
Fact section of a website. Inclusion of assessment data in such straightforward and 
user-friendly formats may foster greater interest in and awareness of assessment find- 
ings (see vignette 3 & 4). 


Use of Evidence of Student Learning 

A little over a third (34%) of the institution websites reviewed provided evidence or Pe — 
examples of the use of student learning outcomes results. Institutions used assessment 

results in a variety of ways, such as to identify areas of improvement, to complete 

regional accreditation self-studies, to alter curriculum or policy, and to inform institu- Vignette 3: 


tion decision making (Figure 3.5). Pe Oe ere Watson 


learning were more likely to be related to accountability than to institutional I nstitutional Research home page 
improvement. provides a changing Fact of the 


Day drawn from the institution's 


¢ The examples found on institution websites of use of evidence of student 


¢ Regional accreditation self-studies (54%) were the most prevalent use of 


assessment results—not a surprising finding, given the important role that assessment data as well as other 
accreditation plays in institutional assessment activities (Kuh & Ikenberry, institutional studies to update 
2009; Provezis, 2010). visitors on selected finding. http:// 
¢ Identifying areas for improvement or potential problems (31%) were the www.cwru.edu/president/cir! 
second most frequent use of assessment evidence. cirhome.html 
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Figure 3.5. Percentage of Institutions Posting the Use of Evidence of Student Learning by Type of Use 


The finding from institution websites that the top two uses of assessment results 
addressed accountability through regional accreditation and improvement reinforces 
the belief that assessment is undertaken for these dual purposes: regional accreditation 
and institutional improvement (Ewell, 2009). However, the reviewed institutions did 
use assessment results for a variety of other purposes as well: 


¢ Curricular uses included examples of modifications to general education 


curriculum (27%) and departmental/program requirements or changes 
within specific courses (18%). 


¢ — Several institutions (15%) used the evidence of student learning to make 
changes in the assessment process itself, such as employing different measures 
or tests, while 10% of the institutions used the results to compare student 
performance on assessment activities with other schools. 


e Although evidence confirming these claims was difficult to locate on the 
institution websites, 29% of the institution websites claimed that evidence 


of student learning was used in their strategic planning process. 
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¢ Few institutions used evidence of student learning to alter institution policy 
or to determine resource allocation (13%). 


e Some institutions used evidence of student learning to determine that they 
needed to gather more data prior to making a decision or to determine if 
certain results were problematic (12%). 


¢ Only 3% of the institutions claimed they were satisfied with the results and 
used them to either market their institution or to show external stakeholders 
they were doing a good job educating students. 


Of the institution websites that presented evidence of the use of student learning 
outcomes results (Figure 3.6), 42% showed only one instance or type of use. The 
remaining 58% of institutions showed multiple types of use, with 1% of the institu- 
tions showing seven different types of use. 


The websites of California State University, Los Angeles, as well as San Francisco State 
University provide a template for the use of evidence of student learning by asking 
departments to report the many ways in which use occurs (see vignette 5). 
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Figure 3.6. Percentage of Institution Websites Showing One or More Types of Use of 
Evidence of Student Learning 


¢ The institutions that showed one type of use mainly used evidence of student 
learning for regional accreditation in the self-study (54%). Other uses 
included department- or course-level use (13%), or identification of areas of 
institution-level improvement (11%). 


¢ The remaining uses (22%) were vague in language, meaning the institu- 
tion website claimed that evidence of student learning was used in decision 
making, planning, or improving outcomes and learning—without any clear 
indication or examples of such use beyond the claim. 


¢ — Institutions that provided multiple examples of the use of evidence of student 
learning were very diverse in their application, meaning that no single area of 
use was prominent. 


Conclusion 


Slightly over half of the institutions in the study (54%) posted results of student 
learning outcomes assessment on their website, suggesting that institutions are begin- 
ning to provide access to assessment results. At the same time, only a third (34%) of 
institutions shared examples of how they were using results. As institutions become 
more proficient and confident in reporting assessment results, the opportunity and 
challenge of showing how evidence is used is likely to grow. Institutions currently 
display a large amount of “consumer information” in the form of performance indica- 
tors (78%), but these indicators are not directly related to student learning. 


Vignette 4: 


Westmoreland Community 
College provides a tease to the 
full CCSSE results on the main 
institutional research office page 
under a changing Interesting 
Fact section. http://wece.edu/ 
IE.aspx?PID=420 
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Genuine transparency is more than simple disclosure. Most of the results posted 
on institution websites reviewed in this study were not presented in an easy-to-read 
form or tailored to a specific audience. Few institutions (10%) discussed next steps or 
implications of the results. Most of the reports were lengthy, lacked a summary, and 
contained too much information for the average person to review. 


While the majority of institutions used the results of student learning for self-study 
(54%), even more institutions (69%) did not state any use of results for improve- 
ment of student learning. Only 3% of the institutions claimed they were pleased with 
the results, yet only 31% identified areas for improvement. No institution examined 
in the study provided information suggesting that the changes made based on assess- 
ment results had led to enhanced learning. Furthermore, most examples of the uses of 
assessment data tended to be vague or limited to departmental contexts as opposed to 
institutional changes. 


On a more positive note, institutions do appear to be making strides in the public 
display of assessment results and how they use this information. Examples of institu- 
tions that are working on reporting and using results have been presented throughout 
this report, and additional examples of such are provided below. 


Even so, use of evidence of student learning remains largely a function of regional 
accreditation self-study. Assessment results are mostly still found on internal institu- 
tional research web pages only and, when shared, they are difficult to comprehend or 
apply. Much more can be done to make assessment information easier to understand 
and to use by both external and internal audiences. As institutions continue to develop 
assessment initiatives it will be important to pay more attention to the presentation 
and use of their results. 


Instructive Examples of Reporting Student Learning 
Outcomes Assessment Online 


In addition to the examples listed in the text, the following provides additional 
resources that may be useful to an institution that is looking for examples of good 
practices: 
¢ The Community College of Allegheny County provides PDF copies of news- 
letters on specific learning outcomes results on their website. Four to five 
pages in length, these newsletters clearly outline the findings, identify areas 
of improvement, and provide next steps. Further, the reports are composed 
in clear, straightforward language and utilize graphs, highlighted tables, and 
selected results. http://www.ccac.edu/files/PDF_Document/0246280caf704 
c8a90cdb1dc70dcl fa3.pdf 


¢ The University of California—Irvine provides on its website an example of 
a report tailored to the Council on Educational Policy. This example pres- 
ents results from a study of summer courses linking findings from course 
evaluations, surveys, interviews, and subsequent course grades to present 
a coherent picture of the quality of summer instruction. Winnowed down 
to three pages from the full 25-page report, this report summary is for a 
specific audience and includes conclusions, five recommendations, and 
student learning outcomes findings. http://www.assessment.uci.edu/reports/ 
documents/Conclusions_and_Recommendations_Summer_Instructional_ 
Quality.pdf 

¢ American InterContinental University provides a link at their website home 
page to an outcomes page presenting selected information to multiple audi- 
ences in easy-to-read bullet lists and graphs of the responses of students, 


alumni, and employers to multiple assessment and surveys. http://www. 
aiuniv.edu/Student-Life/Outcomes/AIU-Online 


¢ Dominican University of California provides a different means for external 
audiences to access multiple types of performance-indicator-related infor- 
mation across their institution website by providing a three-page table of 
required disclosure information along with the link to that information. 
http://www.dominican.edu/about/current/files/consumerinformationsum- 
mary.pdf 


Vignette 5: 


California State University, Los 
Angeles as well as San Francisco 
State University use a reporting 
template that asks each program 
and department to respond to a 
series of questions including: 


° Are there formal learning 
outcomes? 


° Where are they published? 


© Other than GPA, what data 
are used including direct and 
indirect assessments? 


© What is the process and 
frequency of data collection? 


¢ Who interprets the evidence? 
¢ How are the findings used? 
° Date of last program review 


© What were results of most 
recent student assessment? 


This template provides, in 
one location, information on 
the data collection and use of 
assessment activities across 
departments and programs. 


https://sites.sfsu.edu/sites/ 
sites.sfsu.edu.air/files/ 
Report%20Due%20Dates%20 
%20Educational%20 
Effectiveness%20Indicators.pdf 


http://www.calstatela.edul/ 
academic/aa/assessment/ 
AssessmentReports/Educational 
Effectiveness WASCtable06-07. 


paf 
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Section 4 


Online Transparency of Student 
Learning Outcomes Assessment 


The emergence of the Internet in the early 1990s radically changed how 

information is disseminated and accessed. One demonstration of this is in how 

colleges and universities use the Internet for their communications through both 

college email systems and institution websites. College and university websites 

now provide information for a variety of stakeholders including prospective ee 

students and families, current students, faculty, staff, and alumni. Whether one 

is looking for an institutions application for admission, its policies for tenure, or Whether for reasons of 

a schedule of its athletic events, the college website is the central location to find ape 
a ae ee accountability or for reasons 

this disparate information and much mote. Yet, despite the online availability of 


so much information from colleges, observers have argued that higher education of integrity, as the use of and 
institutions are not providing access to meaningful information on student expectations from the Internet 
learning (Commission on the Future of Higher Education, 2006). continue to grow, institutions 


will be called to place more and 
more information about student 
learning and quality assurance on 
their websites. 


Why should institutions post information about student learning on their 
institution web pages? The reason is twofold. First, the information on the 
websites can tell various external stakeholders (such as students, families, and 
alumni) what students learn at the institution and what the institution is doing to 
assure quality. Second, the information can signal to internal stakeholders (such 
as faculty, staff, and administrators) that their assessment efforts are valued and 
can provide these audiences with the data to make decisions and improve student 
learning. Third, through online communication of assessment information, 
institutions can meet a higher level of accountability. From the perspective of 

the regional accrediting agencies, for example, making information on student 
learning public is a matter of institutional integrity (Provezis, 2010). Whether for 
reasons of accountability or for reasons of integrity, as the use of and expectations 
from the Internet continue to grow, institutions will be called to place more and 
more information about student learning and quality assurance on their websites. 


Given that a college’s central function is education, one may reasonably ask: What 
information is publicly available about student learning and the assessment of that 
learning on the college website? Researchers at the National Institute for Learning 
Outcomes Assessment (NILOA) examined college and university websites to learn 
what assessment information was being made available. The research involved 
looking at nearly 1,000 institution websites of undergraduate degree-granting 
2-year and 4-year institutions (for details on the methodology, see Appendix A). 
‘The goal of studying institution websites was to find out if institutions posted 
assessment information, what information they posted and where, and what 
factors possibly influenced the posting of assessment information. 


Given the complexity and somewhat disparate questions of the research project, 
the findings were organized into three reports which may be viewed individually 
or asa full report: Revealing Student Learning Outcomes: National Transparency 
Initiatives Make a Difference, discusses the role of the national transparency 
initiatives on the assessment information posted; The Changing Landscape of 
Transparency in Student Learning Outcomes Assessment, compares and contrasts the 
study of websites done in 2009 with the one completed in 2010; and Presenting 
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and Using Student Learning Outcomes Assessment Results, focuses on the assessment 

results that were found and the ways that institutions use them. Summarizing 

these three reports, this overview outlines the major factors associated with 

institutional transparency; the audiences that the posted assessment information 

appears to address; and the amounts and types of information found on the 

websites, specifically looking for student learning outcomes statements, assessment SSS — 
activities, evidence of student learning, and the usez of student learning evidence. 

Finally, this overview offers a means for institutions to address transparency 


through using the NILOA Transparency Framework as a tool. Calls have been made for 
institutions to show more 
Factors Influencing Transparency information about what students 


learn, and several factors appear 


Calls have been made for institutions to show more information about what . Aenea: 
to influence institutions to do so. 


students learn, and several factors appear to influence institutions to do so. While 
the NILOA report Revealing Student Learning Outcomes: National Transparency 
Initiatives Make a Difference discusses these findings in more detail, some of the 
highlights from that study are presented here. 


Four national transparency initiatives created by several national higher 
education organizations and used by member institutions to present information 
about students and on institution performance were examined: the University 
and College Accountability Network (U-CAN), the Voluntary System of 
Accountability (VSA), Transparency By Design (TbD), and Achieving the 
Dream (ATD). (For full descriptions of each of these initiatives, see Table 1 in 
Revealing Student Learning Outcomes). Participation in these initiatives appeared 
to have an impact on institutions’ efforts to be more transparent. An analysis of 
institutions participating in these initiatives versus those not participating showed 
that institutions participating in the initiatives not only provided assessment 
information on their respective initiative web pages, but they also provided more 
such information on their own websites. 


Institutional control—public vs. private/independent—was found to be related 
to the extent to which institutions show assessment information on their websites 
(see Revealing Student Learning Outcomes). Public institutions, more so than 
independent institutions, described their assessment activities, posted assessment 
results, and discussed the use of results more frequently than did independent 
institutions. Of course, many public institutions are under state mandate to 
provide information, so this requirement may have influenced the information 
posted (For a more detailed description of state policies and institutional 
assessment activities, see Ewell, Jankowski, & Provezis, 2010). Private institutions 
may not have undergone the same type of scrutiny from state oversight; therefore, 
this could be a factor as to why less information was found on these institutions’ 
websites. 


The regional accreditation process appears to be an impetus for institutions’ 
participation in institutional assessment activities and, as such, accreditation has 
advanced collegiate assessment on many campuses (Jankowski & Makela, 2010; 
Kuh & Ikenberry, 2009). While none of the regional accreditation organizations 
specifically require institutions to post assessment information (Provezis, 2010), 
many institutions have created an accreditation page on their websites that 
includes assessment information and are incorporating assessment information 
throughout their web pages. Specifically, those institutions accredited in the last 
three years were found to be more likely to post their student learning outcomes 
statements, assessment activities, assessment results, and examples of how they 
were using evidence gained from student learning assessment than institutions not 
accredited recently. 
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Audiences for Assessment Information 


‘There are two primary audiences for assessment information: those internal to the 

campus community and those external to it (Ewell, 2009; Suskie, 2009; Volkwein, 

2008; Walvoord, 2004; 2010). In his NILOA Occasional Paper, Assessment, ESS, — 
Accountability, and Improvement: Revisiting the Tension, Ewell (2009) wrote that 

the assessment process and information gathered from it is usually different 


for internal audiences than it is for external audiences. Internal audiences seek The NILOA studies of websites 
assessment information that is ongoing rather than finite and that can be used for found that most assessment 
improvement. External audiences, on the other hand, seek assessment information information was on the web pages 


that is more compliance driven and that can be easily reported (for example, a geared to internal audiences, 


such as the offices of institutional 
research and chief academic 


test score). Regardless of the intent, institutions can communicate assessment 
information that both informs internal audiences and responds to external 
audiences if the information is centrally located and meaningfully presented with 
clear language and graphics. Instead, however, the NILOA studies of websites officers. 
found that most assessment information was on the web pages geared to internal 

audiences, such as the offices of institutional research and chief academic officers 

and, furthermore, that most of this information was difficult to understand for 

someone unfamiliar with assessment. First described in our original study on 

websites (Exploring the Landscape: What Institutional Websites Reveal About Student 

Learning Outcomes Activities), these findings were confirmed in our most recent 

study (The Changing Landscape of Transparency in Student Learning Outcomes 

Assessment). Given this finding continues to emerge, more can and should be done 

to make assessment information accessible to the different audiences seeking it. 


Student Learning Outcomes Statements 


The skills, knowledge, attitudes, and competencies that students should acquire 
as a result of their college educational experiences should be embodied in 

student learning outcomes statements. Most colleges indicate that they have 
such statements at the institution level, as evidenced by two thirds of the chief 
academic officers surveyed in the NILOA National Survey (Kuh & Ikenberry, 
2009). A reasonable expectation following from that finding is that institutions 
would post these statements on their websites, and yet only 52% of institutions 
were found to do so. Even when statements were posted, they were often difficult 
to find because they were embedded in larger reports or documents (see The 
Changing Landscape of Transparency in Student Learning Outcomes Assessment). The 
difference between the number of institutions with statements and the number 
who post them suggests that institutions have this information but the effort 

was not made to make it available. In other words, colleges have created student 
learning outcomes statements—possibly for accreditation or strategic planning— 
but they are not communicating them widely enough. 


Assessment Activities 


Chief academic officers purport that their institutions engage in more assessment 
activities than are represented on their institution websites (Jankowski & Makela, 
2010). From the time of the original review of institution websites, in 2009, until 
the review a year later, in 2010, NILOA researchers found that within just one 
year institutions had begun to show more information. Even so, they were still not 
showing as much as they were claiming to have done (see The Changing Landscape 
of Transparency in Student Learning Outcomes Assessment). While assessment 
activities may be in place, the mechanism for communicating about these 
activities may be underdeveloped or misunderstood. The Internet provides the 
means to post assessment information, but an institution needs to decide what to 
show and how to show it. Given the call for more information, institutions must, 
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at the very least, acknowledge the work they are doing in this area. Ideally, for a 
fuller picture of learning at the institution, this information should include both 
direct and indirect assessment information. To date, however, the most prominent 
information found on institution websites are results of surveys—indirect 
measures of learning. Granted, surveys do provide data that, if presented well, can 
be easily digested by multiple audiences. Even so, the demands placed on colleges 
require more substantial, robust information about what students actually learn at 
an institution. By not providing the right amount of information, institutions are 
falling short of meeting demands for greater transparency. 


Evidence of Student Learning 


While having stated objectives and a means for assessing are important steps 
toward understanding student learning, more information is needed to assure that 
learning is actually taking place. Institutions can show such evidence of student 
learning by presenting results from their assessment activities. Most institutions 
by now should have results of some kind. The NILOA research sought to see if 
those results were available via institution websites, and the report Presenting and 
Using Student Learning Outcomes Assessment Results explains that in many cases 
institutions were reporting results of learning outcomes assessment information 
on the institution web pages (see Presenting and Using Student Learning Outcomes 
Assessment Results). Nearly 80% of institutions included performance indicators 
such as graduation rates, placement rates, licensure pass rates, fact books and so 
on, while 57% posted results such as survey findings and standardized tests. Even 
so, this information tended to be difficult to find and was often buried in longer 
reports that lacked a summary of the findings. Given the difficulty of finding the 
documents containing this information, it is unlikely that average Internet users 
off campus would search for this information. Furthermore, it is doubtful that 
more than a handful of people on campus would be aware of the information 
either. By obfuscating the results, institutions are not promoting their use. 
Instead, the institutions must prioritize making the information easily found and 
understood. 


Use of Evidence of Student Learning 


Explaining how assessment activities are being used provides the public with an 
idea that institutions are continuously examining the quality of the education 
they provide and making decisions that lead to improvement. Yet, in describing 
how assessment information was being used, only about one third (34%) of the 
institutions provided examples of use. As expected, most institutions stated that 
the results were used for improvement, to complete regional accreditation self- 
studies, to alter curriculum or policy, or to inform institution decision-making. 
More than half of the institutions said that the assessment results were used 

for institution self-study, whereas only about 30% of institutions were using 
assessment for improvement. At this point, none of the regional accreditation 
organizations requires complete transparency of the accreditation process and 
reports. While it may not be necessary to provide complete transparency, a 
worthwhile venture for institutions would be to provide reflections on lessons 
learned through the self-study process. Assessment highlights could also be 
featured regularly on a website. 


Next Steps 


The Internet provides a useful platform for communicating information in a 
timely manner. The NILOA website research found that while institutions post 


assessment information more frequently than many would likely have guessed, the 


information is often hidden or not presented effectively. The NILOA research also 
found that factors associated with institutions’ showing information online are 


While having stated objectives 
and a means for assessing 

are important steps toward 
understanding student learning, 
more information is needed to 
assure that learning is actually 
taking place. 
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their participation in national transparency initiatives, institutional control, and 
the accreditation process. The three NILOA reports Revealing Student Learning 
Outcomes: National Transparency Initiatives Make a Difference; The Changing 
Landscape of Transparency in Student Learning Outcomes Assessment; and Presenting 
and Using Student Learning Outcomes Assessment Results offer a more in-depth look 


at these finding. SS 


For reasons of institutional integrity and accountability and because the desire 
among higher education stakeholders will surely increase to know more about The NILOA website research 


what institutions are doing to assure they are providing a quality education, found that while institutions post 
institutions should consider ways to better communicate assessment information ssestnent inform WiDinOre 


website is NILOA’s Transparency Framework, which is available on the NILOA frequently than ee would likely 
website at https://www.learningoutcomesassessment.org/ourwork/transparency- have guessed, the information 
framework. The NILOA Transparency Framework not only provides ways to is often hidden or not presented 
improve web-based communications but also points to examples of institutions effectively. 

already working toward web-based transparency. The Transparency Framework is 


on their own websites. One recommended tool for examining an institution's 


organized in six categories representing the six components of student learning 
outcomes assessment: student learning outcomes statements, assessment plans, 
assessment resources, current assessment activities, evidence of student learning, 
and use of student learning evidence. Users of the Transparency Framework 
can evaluate their own institution websites or develop an assessment website 

by reviewing each category to see if that information is available and how it is 
being presented on the institution website. By using this framework as a guide, 
institutions can make a focused effort to communicate assessment information 
both for external accountability and for internal improvement. 
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Appendix: Web Study Methodology 


The 2010 NILOA Web Study sought to document the impact of national transparency initiatives on institutional 
learning outcomes disclosure through an examination of institution websites for evidence of assessment activity, public 
disclosure of results, and examples of the use of assessment findings. A team of NILOA researchers systematically 
reviewed 200 institution websites from July 2010 to September 2010 over a period of 120 hours to answer the following 
guiding questions: 


1. What and how much information do institutions display on their website regarding student learning 
outcomes assessment? 


2. Which institution web pages provide information about assessment of student learning outcomes? 


3. To what extent do institutions show results and describe their use of the results on their institution web 
pages? 

4. What and how much information do institutions display on national transparency initiative websites 
regarding student learning outcomes assessment? 


5. Is there a difference in learning outcomes disclosure on websites between institutions involved in national 
transparency initiatives and institutions not involved in those initiatives? 


The results of this research are presented in a series of reports from NILOA on the web study findings. 


The Web Study Sample 


The sample for the web study was composed of two sets of institutions: 1) 100 institutions involved in a national 
transparency initiative and 2) 100 institutions not involved in a national transparency initiative but similar in size, 
control, setting, type, and regional accreditation to the institutions involved in national transparency initiatives. The 
national transparency initiatives with which the first set of 100 randomly selected institutions was involved included the 
Voluntary System of Accountability (VSA), Achieving the Dream (ATD), the University and College Accountability 
Network (U-CAN), and Transparency by Design (ITbD). The population was determined by visiting each national 
transparency initiative website and adding each institution listed into a database. The entire population of institu- 
tions involved in any of the above national transparency initiatives was 1,245. Those institutions were then cleaned by 
removing several that were not regionally accredited or undergraduate degree-granting postsecondary institution in the 
U.S. This list was also checked against the 2009 NILOA National Survey list to ensure that the institutions included 
were part of the larger NILOA study population. ‘The final institution population included 1,196 eligible institutions. 


To determine the sample of national transparency initiative websites to scan, 100 institutions were selected from the 
national transparency initiative population starting with those that were part of the 2009 NILOA Web Study. Involve- 
ment in national transparency initiatives by these 100 institutions was relatively evenly distributed across the four 
national transparency initiatives, except for the Transparency by Design initiative, for which all of the involved institu- 
tions were selected because of their small number (n=14). The sample of national transparency initiative institutions is 
shown in Table Al, below. The 100 institutions not involved in national transparency initiatives were selected for the 
sample based on their matching one of the institutions involved in a national transparency initiative. 


Table Al. National Transparency Initiative Institution Sample 


National Transparency Initiative Institution 
Sample (n) 


Voluntary System of Accountability (VSA) 


Achieving the Dream (ATD) 
University and College Accountability Network (U-CAN) 
Transparency by Design (TbD) 


The Web Study Procedure 


This web study had three components: an institution-wide web page scan, a national transparency initiative web page 
scan, and a gathering of institutional student learning outcomes statements. Researchers first scanned a designated set 
of web pages for each institution, beginning with seven institution-wide web pages: 
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¢ home 

¢ admissions/prospective students 

¢ institutional research office/institutional effectiveness office 
¢ — provost/chief academic officer/academic affairs 

* — assessment/accountability 

¢ accreditation 

* center for teaching and learning 


Any information related to student learning outcomes assessment activities, results, or use found within five mouse 
clicks' from an initial institution-wide web page was recorded in a database. The information’s location in relation to 
the initial page, measured by the number of clicks to get to that location, was a proxy for the information's degree of 
transparency. In addition to recording the occurrences of student learning outcomes assessment activities, results, and 
use, researchers made notes in an “other” category of unique or unanticipated findings, making it possible to document 
the various assessment activities and results posted on institution websites. 


‘The types of assessment activities documented in this scan included standardized tests, local tests, national student 
surveys, local student surveys, alumni surveys, other surveys, portfolios, capstones, awards for assessment, and faculty/ 
staff development.’ In addition, researchers captured screen shots and document examples from the institution websites. 


Researchers recorded evidence of assessment results and use found on institution-wide web pages in the database as well. 
Results include student learning outcomes assessment data, findings, analysis, as well as use of the data. Institutions 
were marked in the database as showing results if they provided publicly available evidence of institution performance 
as well as indirect and direct evidence of student learning on the institution website. If results and use were found on the 
institution web pages, researchers clearly stated the assessment activity for which the results were posted (such as NSSE, 
CLA, etc.) and then indicated where the results were found (such as in the self-study, off the institutional research page, 
etc.). In a text box in the database, using keywords from the Results and Use Web Scan Researcher Guide’ prepared for the 
study, researchers provided structured answers to the following three questions: 


1. Did the institution present data or information on assessment results? 
2. How are the assessment results presented? 


3. How does the institution use the results of student learning outcomes assessment? 


In the second part of the web study, researchers reviewed the website of the national transparency initiative in which 
each institution was involved, if the institutions were involved in one. This meant, as an example, visiting the VSA 
website and recording in the study database whether there was information on student learning outcomes assessment 
activity for a specific institution at the website and, if so, whether results were posted there. 


The third part of the web study procedure involved researchers gathering institution-wide student learning outcomes 
statements from each institution website. Information was recorded in the study database as to whether the statements 
were easily found on the website or were hidden within documents and whether the statement’s language clearly related 
to student learning outcomes. 


Reflection and Reviewer Information 


Each section of the web study database form for researchers included an area for comments and additional items related 
to learning outcomes. Also, the researcher’s initials, the date the web scan was conducted, and the length of time taken 
to complete the scan were recorded because each website was visited at a specific moment in time and was likely to be 
updated after the researcher scanned the site. Researcher training, consistency, and communication were facilitated via 
practice web scans, training sessions, periodic reflective journaling, midterm data analyses, and group feedback sessions. 
Reflective journaling was done at the completion of every 20 scans or at the end of every week, whichever came first. 
This allowed the researchers to think back and comment upon the study procedure in general as well as to reflect on any 
trends or gaps found while scanning the websites. 


"Marketing research has suggested that Internet users are not likely to look for information beyond three to four mouse clicks on a website (Geissler, Zinkhan, 

& Watson, 2006). To ensure generous and complete review in this study and to allow for deeper exploration of the site, five clicks were used for this web scan 

procedure. 

See Jankowski and Makela’s Exploring the Landscape: What Institutional Websites Reveal About Student Learning Outcomes Assessment Activities (2010) for defini- 
tions of each of these activities: https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/WebScanReport.pdf 

> Please contact njankow2@illinois.edu if you would like a copy of the Results and Use Web Scan Researcher Guide. 


2 
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Data Analysis 


Cleaning of the data prior to data analysis involved making sure there were no missing data, reviewing the comments in the 
reflections for any potential analysis issues, and reviewing identified analysis issues that had been updated throughout the 
scanning process. [wo institutions were removed from analysis as the regional accreditation status of one was withdrawn 
and another institution was closed. To keep the number of national transparency initiative institutions and noninitiative 
institutions at 100 each, these two institutions were replaced in the sample with similar matched institutions. Descriptive 
statistics for the combined 200 institutions as well as for each set of 100 institutions were run in SPSS, as were Chi-square 
analyses between the matched groups and within the initiative population. Analyses were run to examine the significance 
of control, institution type, accreditation region, and date of last accreditation on public disclosure of student learning 
outcomes information. Text from the database’s results and use boxes as well as from the national transparency initiative 
website comment boxes were coded and analyzed for themes. 


‘The starting point in coding the text was the list of descriptive terms and questions to consider when looking at posted 
institutional use and results. This list was developed from the literature on effective reporting, including such sources 
as Ketcheson (2001), Middaugh (2010), Suskie (2009), Volkwein (2010), and Walvoord (2004; 2012). Coding of the 
written comments was initially done using the terms given in the study’s guide, and then additional codes were developed 
as needed based on the frequency of a comment or the inability of the then-current list to capture the comments recorded 
by the researchers. Once an initial coding of text was complete, codes were reviewed for clarity, for ability to capture 
comments, and for possible redundancies. Changes were made on the second review and each comment was read again 
to ensure that the coding fully captured the comment coherently. 


Limitations 


One limitation of this study stems from the research team’s knowledge of and experience with navigating institution 
websites. Compared to average users of the Internet, this research team probably had a better idea of the location of 
information and how to search for it, potentially finding more paths to information and generally more assessment infor- 
mation in five clicks than the average user would. 


As a second limitation of this study, the sample of institutions and the national transparency initiative population is not 
representative of the population of regionally accredited undergraduate degree-granting institutions in the U.S. as the 
institutions involved in these national transparency initiatives are more likely than is the larger population of institutions 
to be private, baccalaureate, and master’s institutions. This may be reflective of the institutions involved in the initiatives 
themselves or of the types of initiatives—such as U-CAN, which is specifically for private institutions. 


Another limitation of this study concerns the medium of the website itself, which has the potential to be continuously 
changed and updated. A website scan captures a snapshot of a website at one moment in time and may not necessarily 
reflect practice at a later time. Scanning institution websites gathers information that may partly reflect the importance 
an institution places on the Internet as a means of communicating information as well as the institution’s financial and 
technological capacities. By looking at specific pages and predetermined content criteria, this study addressed this limita- 
tion to some extent. Finally, inter-rater reliability was a concern in this study, as in any project with multiple researchers, 
and steps were taken to address it. First, before researchers began to gather data, they received training on how to scan. 
Second, the researchers’ periodic sharing of reflections provided a means to regularly discuss data-gathering experiences. 
Finally, feedback sessions, meetings, and discussions provided additional contexts for training. 
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